Determining locations of electro-optical pens

ABSTRACT

Determining locations of electro-optical pens. An example apparatus includes a distance determiner to determine a distance of a pen to a scribing surface based on a first image output by a first imager of the pen and a second image output by a second imager of the pen, the first image associated with a first plane of view and the second image associated with a second plane of view; a tilt orientation determiner to determine a tilt orientation of the scribing surface relative to the pen; a corrector to adjust at least one of the first plane of view or the second plane of view based on the tilt orientation of the scribing surface; and a location determiner to determine a location of the pen relative to the scribing surface based on the at least one of the adjusted first plane of view or the adjusted second plane of view.

RELATED APPLICATIONS

This patent arises from a continuation of U.S. patent application Ser.No. 16/481,642, now U.S. Pat. No. 10,936,089, which was filed on Jul.29, 2019. U.S. patent application Ser. No. 16/481,642 is a nationalstage application under 35 U.S.C. § 371 of International PatentApplication No. PCT/US2017/060597, which was filed on Nov. 8, 2017. U.S.patent application Ser. No. 16/481,642 and International PatentApplication No. PCT/US2017/060597 are hereby incorporated by referencein their entireties. Priority to U.S. patent application Ser. No.16/481,642 and International Patent Application No. PCT/US2017/060597 ishereby claimed.

BACKGROUND

A digital pen or smart pen is an input device that captures thehandwriting or brush strokes of a user and converts the same intodigital data that can be utilized by various applications. Camera-baseddigital pens can use digital paper as a writing surface. The digitalpaper is used to facilitate location detection of the digital pen on thewriting surface.

Digital paper can include printed dot patterns that uniquely identifyposition coordinates on the paper. For example, a printed dot patterncan include microscopic marks that are associated with the writingsurface of the paper, a display, or a whiteboard and indicate thelocation of the pen's nib. These location marks can be captured inreal-time by the camera as the pen's nib traverses the writing surface.The digital pen records the dot patterns associated with the pen's nibover time and uploads the patterns to a pen based computer, which candetermine the location of the pen's nib over time and communicate thisinformation in real-time to a host with a display based computer forgraphic rendering or digitally store this information for latercommunication to a computer based graphic rendering system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an example operating environment of a pen locationdeterminer.

FIG. 2 depicts example parts of an electro-optical (EO) pen.

FIG. 3 illustrates an example of orientation of first and second imagersand the overlapping fields of view of the imagers.

FIG. 4 illustrates an example of a first and a second pixel footprint ofa first and second imager on a scribing surface.

FIG. 5 illustrates factors involved in an example distance measurementfor tilted plane encoded dots using triangulation.

FIG. 6 depicts example components of an EO pen location determiner.

FIG. 7 depicts a flowchart representative of computer readableinstructions that may be executed to implement the operations of anexample method to determine the location of an EO pen.

FIG. 8 depicts a flowchart representative of computer readableinstructions that may be executed to implement the operations of anexample method to determine the distance from an EO pen to a scribingplane at a plurality of points.

FIG. 9 depicts a flowchart representative of computer readableinstructions that may be executed to implement the operations of anexample method to determine the tilt orientation of a scribing planerelative to an EO pen.

FIG. 10 depicts a flowchart representative of computer readableinstructions that may be executed to implement the operations of anexample method to determine the three dimensional orientation of ascribing plane.

FIG. 11 depicts a flowchart representative of computer readableinstructions that may be executed to implement the operations of anexample method to correct the plane of view associated with a first orsecond perspective.

FIG. 12 is a block diagram of an example processor platform capable ofexecuting the instructions of FIGS. 7-11 to implement the EO penlocation determiner of FIG. 6.

FIG. 13 is a block diagram of an example processor platform capable ofexecuting the instructions of FIGS. 7-11 to implement the EO penlocation determiner of FIG. 6.

Wherever possible the same reference numbers will be used throughout thedrawing(s) and accompanying written description to refer to the same orlike parts.

DETAILED DESCRIPTION

Example electro-optical (EO) pens described herein utilize a pluralityof sub-miniature digital camera imagers. In some examples, the imagershave a global shutter and synchronization capability that enables the EOpens to capture and decode three-dimensional information from a scribingsurface (display, whiteboard, paper, etc.) at high translational ratesof motion of the EO pen. Thus, some example EO pens can determine withhigher accuracy, using lower positional encoding pattern areal density(resulting in approximately 50 percent more transparent displays) andwith significantly reduced computational overhead, the tilt orientationof the EO pen relative to a position encoded scribing surface. Thisenables more efficient and higher rate decoding of positional encodingpatterns on the EO pen scribing surface. The sub-miniature digitalcamera imagers provide example EO pens with dynamic (real-time)stereoscopic three dimensional (3D) imaging capabilities. Stereoscopic3D imaging presents two offset images that can be combined to give theperception and measurement of 3D depth with high resolution. Otheradvantages of the example EO pens include but are not limited toincreased depth-of-field (DoF), pen shaft/nib centration in the penbarrel and the capacity to determine an EO pen's full rotational andhover orientation relative to a scribing surface. Also, in someexamples, the need to accommodate a fold mirror to relay images capturedfrom a lens to a digital imager, is eliminated. Thus, examples describedherein significantly enhance the operational performance of EO pens.

In some examples, the position encoded data patterns are positionedrelative to a reference grid, and enable the EO pen to determine thelocation of the nib of the EO pen on a scribing surface. An example EOpen uses a single digital camera to capture images of the location ofthe pen's nib on the scribing surface relative to the position encodedpattern. Depending on the pattern type that is used, a removal ofperspective distortion from images when the EO pen is tilted can becomputationally difficult and very time intensive using single cameramethods. Such methods include vanishing point line-projection fittingand the associated iterative convergent calculations. However,perspective vanishing point calculations become decreasingly accurate asthe tilt orientation of the pen becomes larger. Other approaches useFourier image transform methods to extract highly graphically dense dotpattern image content located on or adjacent to principal orientationand encoding axes which are extracted by the computationally intensetransforms. These methods may have included hardware dedicated digitalsignal processing (DSP) methods to extract information for singleimages. Providing second camera stereographic information relative tothe graphically encoded scribing plane (paper, display or whiteboard)computationally simplifies and renders more accurate the determinationof the tilt and rotational orientation of the EO pen relative to thescribing surface as described herein.

The computational methods that are used by single camera EO pens may bepaired with either highly bounded positional encoding marks (exactgeometric location known for a large set of marks/dots) or asignificantly higher density of encoding marks on the scribing surface(paper or display). For example, in one approach, an EO pen encodingpattern uses one fixed mark for each variable position encoded mark toaccommodate calculations used to rectify perspective distortion in theimage from dynamically varying pen tilt as the pen user moves the EO penacross the scribing surface. In some examples described herein, areduction in the areal density of encoding marks by approximately fiftypercent or more is enabled by the elimination of most if not all fixedposition alignment marks. This results in display films with encodeddots/marks that are more transparent. Such display films are moreaesthetically and visually functional. Some examples also provide tiltand rotational orientation data with accuracy to sub 1 degree anglesbetween the scribing surface and the EO pen. This enables improvedpositional transformation of perspective distorted images of thepositional dot data that is used to decode reference grid decodabledata.

It should be appreciated that because two imagers are used, in someexamples, one of the imagers will have a reduced depth of fieldobservation orientation relative to the other imager because it has adiffering proximity to the scribing surface. Selective decoding of atilt normalized pattern from the imager with the greater depth of fieldobservation orientation provides improved system performance byproviding increased operational depth of field for the pen (e.g., byensuring the use of the imager that has the best focus range for suchpurposes).

FIG. 1 shows an example operating environment 100 of a pen locationdeterminer. FIG. 1 shows EO pen 101, position encoded pattern 102,non-visible reference grid 103, scribing surface 104, pen pattern 105,light source 107, first imager 109, second imager 111, pen locationdeterminer 113, microprocessor 115, host 116, display 117, receiver 119and displayed writing 121.

The EO pen 101 is an electronic writing and/or drawing instrument thatcaptures handwriting, or other user directed pen strokes (e.g.,patterns), and converts them into digital data. The digital data can betranslated into writing and/or drawings that correspond to the penstrokes 105 that are shown on the display 117. In some examples, the EOpen 101 is used in conjunction with a scribing surface 104. The scribingsurface 104 includes the position encoded patterns 102, which arepatterned markings that identify position coordinates of the EO pen 101on the scribing surface 104. The patterned markings are positionedrelative to a virtual and/or non-visible reference grid 103 and allowthe EO pen 101 to determine where the nib of the EO pen 101 is locatedby decoding the patterned markings location relative to the referencegrid 103.

FIG. 2 is a cross-sectional view of the EO pen 101 and depicts examplecomponents of such. Referring to FIG. 2, in some examples, components ofthe EO pen 101 can include but are not limited to light sources 107, 203and 205, imagers 109 and 111, printed circuit board (PCB) 201, circularPCB 207 and pen shaft opening 209. In some examples, the EO pen 101 caninclude a single light source, e.g., light source 107 as shown inFIG. 1. In other examples, the EO pen 101 can include a plurality oflight sources, e.g., light sources 107, 203 and 205 as shown in FIG. 2.In some examples, the PCB 201 runs the length of the EO pen 101 and cansupport the microprocessor 115, Bluetooth system, etc. In some examples,PCB 207 can be located at the front of the EO pen 101 and can face thescribing surface 104 (FIG. 1). It should be appreciated that thecomponent organization shown in FIG. 2 is an example componentorganization and other component organizations can be used.

As shown in FIG. 1, the EO pen 101 images the scribing surface 104 withimagers 109 and 111. Referring to FIG. 3, in some examples the imagers109 and 111 can be attached to the small diameter PCB 207 that can belocated at the front of EO pen 101. The imagers 109 and 111 face thescribing surface 104 (FIG. 1). As shown in FIG. 3, the imager 109 has afirst field of view 301 and the imager 111 has a second field of view303. The field of view 301 and the field of view 303 overlap to enablestereographic depth determination corresponding to imaged points asdescribed herein below. In addition, the field of view 301 and the fieldof view 303 overlap to produce a plane of view 308 (in FIGS. 3 305 and307 are the plane-of-view for imagers 109 and 111 respectively) on thescribing surface 104 (FIG. 1), within which an EO pen nib 309 istangentially located. As shown, in FIG. 3, the pen nib 309 is located atthe end of the pen stylus 311. The area that is imaged is illuminated bythe light source 107 (FIG. 1). In some examples, the light source 107can include a diode. In other examples, other types of light sources canbe used.

In the FIG. 3 example, the first imager 109 and the second imager 111image the part of the scribing surface 104 where the nib 309 of the EOpen 101 is positioned. The first imager 109, images the scribing surface104 (FIG. 1), from a first perspective, and the second imager 111,images the scribing surface 104 (FIG. 1), from a second perspective. Thefirst imager 109 and the second imager 111 capture tangential images ofthe nib 309 of the EO pen 101 relative to the position encoded patterns102 (FIG. 1) of the scribing surface 104 (FIG. 1). The light source 107(FIG. 1) is used to illuminate the scribing surface 104 (FIG. 1) inorder to facilitate imaging by first and second imagers 109 and 111. Insome examples, the light source 107 (FIG. 1) can be strobed in order toboth capture position encoded pattern 102 (FIG. 1) motion and preserveEO pen battery life.

FIG. 4 illustrates a first 403 and second 405 pixel footprint of a firstand second imager on a scribing surface. Referring to FIG. 4, the firstimager 109 (FIG. 1) and the second imager 111 (FIG. 1) produce,respectively, a first pixel footprint 403 and a second pixel footprint405 of the scribing surface 104 (FIG. 1). The center of the field ofview of the first pixel footprint is 404 and the center of the field ofview of the second pixel footprint is 406. The first pixel footprint 403and the second pixel footprint 405 overlap to define the boundaries of astereoscopic dot-sample plane 401. It should be noted that for FIG. 4the relative positon and overlap of the stereoscopic dot-sample plane401 is for an orientation of the EO pen normal to the scribing surface.As the orientation of the EO pen rotates relative to the scribingsurface the overlap of images defining the dot-sample plane 401 willdynamically change. In some examples, a plurality of dots (e.g., threedots, etc.) in the stereoscopic dot-sample plane 401 is used to rapidlytabulate multiple 3D (x, y and z) coordinates of the stereoscopicdot-sample plane 401 and thereby the orientation (tilt) of thestereoscopic dot-sample plane. In other examples, other numbers ofcorresponding dots can be used to rapidly tabulate 3D coordinates (x, yand z) of the stereoscopic dot-sample plane 401 and the orientation(tilt) of the stereoscopic dot sample-plane 401. In some examples withEO pen orientated normal to the scribing surface, the overlapping partsof the first pixel footprint 403 and the second pixel footprint 405 onthe scribing surface 104 (FIG. 1) can define a 220×220 pixel area. Inother examples with the EO pen orientated normal to the scribingsurface, overlapping parts of the first pixel footprint 403 and thesecond pixel footprint 405 can define other sized pixel areas. Thelocation 407 of the tip or nib of EO pen 101 (FIG. 1) establishes thecurrent point of the pen to dot-plane tilt axial point of rotation. Insome examples, two or more stereoscopic imagers (e.g., cameras, etc.)that have different sized pixel footprints can be used. For example, asshown in FIG. 4, a first field of view 409 and a second field of view411 are associated with first and second imagers that have smaller pixelfootprints than the first pixel footprint 403 and the second pixelfootprint 405.

FIG. 5 illustrates an example measurement of a distance between an EOpen having first and second imagers and a dot located in a tilted planeof encoded dots (that is associated with a scribing surface). Thedistance measurement is performed using triangulation based on first andsecond images that are imaged by the first and second imagers. In theFIG. 5 example, the first and second images are respectively associatedwith similar triangles LL1L3 and LL2P, and RR1R3 and RR2P. Because LL1L3and LL2P are similar triangles:z/f=x/x1  (1)

Similarly, because triangles RR1R3 and RR2P are similar triangles:z/f=(x−b)/x2  (2)

In some examples, for a Y-axis that is perpendicular to the page in FIG.5:z/f=y/y1=y/y2  (3)

For stereo imagers with parallel optical axes, focal length f, baselineb and corresponding image points (x1, y1) and (x2, y2), the location ofthe 3D point can be derived from equations (1), (2) and (3) as follows:Depth z=f*b/(x1−x2)=f*b/dx=x1*z/f or b+x2*zfy=y1*z/f or y2*z/f

It should be appreciated that d is the disparity in the position of animage of a dot in the imager plane of the first imager and the image ofthe dot in the imager plane of the second imager. More specifically, itis the distance between corresponding points when the two images aresuperimposed. The disparity data can be used to determine a disparitymap, which can be converted to a 3D mapping of the imaged point orpoints. In this manner the 3D orientation of the scribing surface can bedetermined. In some examples, the 3D “reconstruction” is performed asdescribed herein with reference to FIGS. 6 and 10. In other examples,other manner of converting the disparity data into 3D data can be used.Because they are not reliant on time intensive single camera basedcomputations, in examples, the computation of multiple dot distancesfacilitate the rapid determination of the 3D orientation of the scribingsurface plane.

Based on the determined 3D tilt orientation of the scribing plane, theplane of view associated with either the first perspective or the secondperspective is corrected. The perspective that is corrected isdetermined based on a comparison of the depth of field associated withthe first perspective and the depth of field associated with the secondperspective. In particular, the plane of view associated with theperspective that has the best depth of field is corrected (e.g.,adjusted). In some examples, the perspective that is deemed to have thebest depth of field is the perspective that is closest to the scribingsurface. For example, the perspective that is closest to the scribingsurface can be determined by identifying the closest point to eachimager on the scribing surface and then comparing the distances to thosepoints to determine which point/imager is closest. Any other manner ofdetermining the perspective that is closest to the scribing surface canadditionally or alternatively be used. The coordinates of the EO pen aredetermined based on the corrected plane of view. In particular, thelocation of the nib of the EO pen on the scribing surface can beaccurately determined because of the ability to precisely determine thelocation of dots based on the perspective corrected distorted imagerelative a reference grid.

Referring again to FIG. 1, the EO pen location determiner 113 implementsthe use of stereoscopic images to determine the coordinates of the nibof the EO pen 101 on the scribing surface 104 as is described herein.Based on the coordinates, EO pen location determiner 113 determinesand/or updates the location of the EO pen 101 on the display 117, suchas by providing inputs to a display scribing application. As a part ofthe determining and/or updating of the location of the EO pen 101 on thedisplay 117, a distance between the EO pen 101 and the scribing surface104 at a plurality of points is determined which provides a plurality ofdistance measurements (e.g., from the plane of view) upon which a tiltorientation of the scribing plane relative to the pen is determined.

In operation, the first imager 109 and the second imager 111 image inreal-time the position encoded scribing surface 104 of either a displayscreen, paper, whiteboard or other future media. As describedhereinabove, the field-of-view of the first imager 109 and thefield-of-view of the second imager 111 at least partially over-lap onthe scribing surface 104. The overlapping portion of the field-of-viewof the first imager 109 and the field-of-view of the second imager 111define a plane-of-view (PoV) that is used to cross-correlatecorresponding dots (e.g., marks, etc.) imaged by the first imager 109and the second imager 111. In some examples, stereographic imageprocessing can be applied on the shared image dots via appropriatecomputation of the distance to the scribing plane from the pen atseveral points to obtain the pen to scribing surface tilt orientation asdiscussed herein. In some examples, the overlapping portion of theseimages change with a pen to scribing surface tilt orientation. Inexamples, sufficient corresponding imager dots are used to perform highaccuracy (many sample) calculations of the pen to scribing surface tiltorientation. In some examples there are more than thirty correspondingimager dots. In other examples, the number of corresponding imager dotscan be more or less than thirty. In some examples, from thesecorresponding imager dots multiple distance measurements can beperformed by logic circuitry (e.g., a microprocessor) of the EO pen 101and can be used to compute the tilt orientation of the scribing planerelative to the EO pen 101. Using the scribing plane tilt orientationinformation the EO pen 101 then corrects the PoV of one of the imagers.In some examples, the imager that is selected for PoV correction ischosen based on a determination that the imager has the bestdepth-of-field as compared to the other imager. In some examples, theimager with the best depth-of-field is determined to be the imager thatis closest to the scribing surface. In some examples, the imager withthe least number of detected errors in the dot pattern is selected forpattern decode. In some examples, the selection depends on theorientation of the pen (and thus the orientation of the imagers) at thetime that the determination is made. Based on the selection of theimager with the best depth-of-field or alternative criteria, the imageis corrected for image perspective distortion that is caused by thedynamically changing tilt orientation of the EO pen 101 relative to thescribing surface 104.

The image that is corrected for image perspective distortion is theimage that is used to determine the location of the dots relative to areference grid. In this manner, the position of the dots can be reliablydecoded to determine where the tip or nib of the EO pen 101 is locatedon the scribing surface 104. In some examples, the position of the dotscan be reliably decoded to an accuracy of plus or minus 15 microns. Inother examples, the position of the dots can be reliably decoded withgreater accuracy. The pen tip/nib location/coordinate information isthen communicated (e.g., via a wireless Bluetooth (or other)communications packet) back to the host computer system 116 (personalcomputer, notebook, Workstation, Tablet, phone, etc.) to be used by adisplay scribing application to update the EO pen 101 location on thedisplay 117, e.g., of the host computer system 116, with high accuracyand low-latency. In some examples, the location/coordinate informationis communicated at 120 hertz or higher. In other examples, thelocation/coordinate information is communicated at other frequencies. Insome examples, for paper scribing surface applications the coordinatescan be stored in a memory of the EO pen 101 for later download to adigital computing system for image or text secondary inclusion indocuments/drawings and/or potentially their further manipulation in aworkflow. In some examples, the imager orientations can be orthogonal(rotated 90 degrees) relative to each other to achieve the smallestdiameter EO pen based on the size of the imagers selected. In someexamples, the images can be manipulated (rotated) relative to each otheras a part of an image processing sequence to obtain epipolar alignment.In other examples the images may not be rotated relative to each otherto obtain epipolar alignment. In some examples, a small foot-print andlow power multi-core vision embedded system-on-chip (SoC) that has beendeveloped for virtual and augmented reality headsets as well as dronevision-based flight control applications can be used.

FIG. 6 is a schematic illustration of components of an EO pen locationdeterminer 113. In the FIG. 6 example, EO pen location determiner 113includes image accessor 601, pen-to-scribing-plane distance determiner603, tilt orientation determiner 605, plane of view corrector 607, penscribing surface coordinate determiner 609 and pen display locationdeterminer 611.

Referring to FIG. 6, image accessor 601 accesses a first image of ascribing surface that is imaged from a first perspective and a secondimage of the scribing surface that is imaged from a second perspective.The first image has a first field of view and the second image has asecond field of view. The first field of view and the second field ofview partially overlap to define a plane of view as described hereinwith reference to FIGS. 3 and 4. Image accessor 601 accesses the firstand the second images from the first imager 109 and the second imager111 in real time. Information from the plane of view provides imageinformation from the scribing surface that is used to determine penlocation.

Pen-to-scribing-plane distance determiner 603 determines a distance froma pen to a scribing plane at a plurality of points. The determining ofthe distance from the pen to the scribing plane at the plurality ofpoints produces a plurality of distance measurements. In some examples,pen-to-scribing-plane distance determiner 603 determines a distance froma pen to the scribing plane at a plurality of points bycross-correlating corresponding dots in the plane of view of the imagesthat are imaged by the first imager 109 and the second imager 111. Insome examples, pen-to-scribing-plane distance determiner 603 usesstereographic image processing techniques to process images of theshared image dots to determine the distance to the scribing plane fromthe pen at several points as discussed hereinabove. In some examples,pen-to-scribing-plane distance determiner 603 determines the distance tothe scribing plane from the pen at several points by determining anoverlap region of the first and the second fields of view, identifyingdots in the overlap region and determining the distance to a pluralityof dots in the overlap region. Pen-to-scribing-plane distance determiner603 uses triangulation to determine the z depth of each of a pluralityof dots. In some examples, pen-to-scribing-plane distance determiner 603can use a formula such as f*b/(x1−x2) to determine the distance z toeach of a plurality of dots, where x1 and x2 are image points associatedwith the first imager and the second imager of FIG. 5. In otherexamples, other formulas can be used to determine the distance z to eachof a plurality of dots. Pen-to-scribing-plane distance determiner 603provides this information to tilt orientation determiner 605 to use todetermine the tilt orientation of the scribing plane.

Tilt orientation determiner 605 determines the tilt orientation of thescribing plane relative to the pen based on the plurality of distancemeasurements z performed by pen-to-scribing-plane distance determiner603. In some examples, tilt orientation determiner 605 determines thetilt orientation of the scribing plane relative to the pen by accessingthe determined dot distances z and determining the three dimensionalorientation of the scribing plane based on the determined distances z.In some examples, three dots are used to calculate the orientation ofthe dot plane to high accuracy (e.g., the distances for three dots inthe plane are used). In other examples, other numbers of dots can beused.

Tilt orientation determiner 605, based on the distance z to each of theplurality of dots, can determine the three-dimensional coordinates ofthe dots using equations such as the following:z=f*b/(x1−x2)=b/dx=x1*z/f or b+x2*z/fy=y1*z/f or y2*z/f

In this example y is the y-coordinate of the three-dimensionalcoordinates of a dot. Moreover, y1 and y2 are image points correspondingto the dot and are associated with a first imager and a second imager,such as the first and second imagers of FIG. 5 (y1 and y2 are not shownin FIG. 5 for purposes of clarity and brevity but are analogous to imagepoints x1 and x2). In other examples, other equations and/or manner ofdetermining the coordinates of the dots can be used.

Tilt orientation determiner 605, based on the coordinates, can determinethe three-dimensional orientation of the dot plane/scribing surface. Forexample, for dots/points P, Q and R having coordinates P(1,−1,3),Q(4,1,−2) and R(−1,−1,1) tilt orientation determiner 605 can determinethe orientation of the dot plane/scribing surface by determining theplane that passes through such points. In some examples, to find theplane passing through points P, Q and R the tilt orientation determiner605 determines the cross product of a vector “a” from P to Q and avector “b” from Q to R to obtain a vector orthogonal to the plane. Inthe preceding example, vector a=<3,2,−5> and vector b=<−5, −2, 3>, thus:

${\overset{\rightarrow}{n} = {{\overset{\rightarrow}{a} \times \overset{\rightarrow}{b}} = {{\begin{matrix}\hat{i} & \hat{j} & \hat{k} \\3 & 2 & {- 5} \\{- 5} & {- 2} & 3\end{matrix}} = {< {\text{-}4}}}}},{16},{4 >}$

Based on the above, the equation of the plane can be determined to be−4x+16y+4z=−32. The manner of determining tilt orientation describedherein above is an example. In other examples, other manner ofdetermining tilt orientation can be used.

Plane of view corrector 607 corrects the plane of view associated witheither the first perspective or the second perspective based on acomparison of a depth of field associated with the first perspective anda depth of field associated with the second perspective and/or based onother criteria such as described previously. In some examples, acorrection algorithm is used. In some examples the correction algorithmcan be implemented in software or hardware or a combination of both. Insome examples, plane of view corrector 607 corrects the plane of viewassociated with either the first perspective or the second perspectiveby accessing the scribing plane orientation information and based onthis information determining whether the first or the second imager hasthe best depth of field or other preferential selection criteria. Theimager that is determined to have the best depth of field is dependentupon the orientation of the EO pen at the point in time at which thedetermination is made. Plane of view corrector 607 corrects the plane ofview using the depth of field of the imager that is determined to havethe best depth of field and/or based on other criteria also describedpreviously. In this manner, plane of view corrector 607 uses theorientation of the dot plane to transform the perspective distortedimage of the imaged dots to a planar undistorted dot field. In someexamples, the imager that is closest to the scribing surface is deemedto have the best depth of field. In other examples, other manner ofdetermining the imager that has the best depth of field can be used. Insome examples, the imager with the least number of detected patternerrors such as smudged or distorted dots in the dot pattern can be used.

Pen scribing surface coordinate determiner 609 determines thecoordinates of the EO pen on the EO pen scribing surface based on thecorrected plane of view. Based on the perspective corrected dotlocations, and the reference grid, the location of the EO pen tip isdetermined. In some examples, grid orientation marks that are imaged byboth imagers provide indicators of directionality and a basis for theincremental separation of superimposed reference grid lines. In someexamples, the coordinates of the EO pen nib/tip location arecommunicated back to the host via Bluetooth wireless connection. Inother examples, other manner of communicating coordinate information tothe host can be used.

Pen display location determiner 611 determines the position of the EOpen on the display 117 based on the pen coordinate informationdetermined by pen scribing surface coordinate determiner 609. In thismanner, the handwriting or brush strokes 121 (FIG. 1) of a user on ascribing surface can be reproduced on display 117.

While an example manner of implementing the EO pen location determiner113 of FIG. 1 is illustrated in FIG. 6, the elements, processes and/ordevices illustrated in FIG. 6 may be combined, divided, re-arranged,omitted, eliminated and/or implemented in any other way. Further, theexample EO pen location determiner includes imager accessor 601,pen-to-scribing-plane distance determiner 603, tilt orientationdeterminer 605, plane of view corrector 607, pen scribing surfacecoordinate determiner 609 and pen display location determiner 611. Theseelements and/or, more generally, the example EO pen location determiner113 of FIG. 6 may be implemented by hardware, software, firmware and/orany combination of hardware, software and/or firmware. Thus, forexample, any of the example imager accessor 601, pen-to-scribing-planedistance determiner 603, tilt orientation determiner 605, plane of viewcorrector 607, pen scribing surface coordinate determiner 609 and pendisplay location determiner 611 and/or, more generally, the example EOpen location determiner 113 could be implemented by one or more analogor digital circuit(s), logic circuits, programmable processor(s),application specific integrated circuit(s) (ASIC(s)), programmable logicdevice(s) (PLD(s)) and/or field programmable logic device(s) (FPLD(s)).When reading any of the apparatus or system claims of this patent tocover a purely software and/or firmware implementation, at least one ofthe example, EO pen location determiner, imager accessor 601,pen-to-scribing-plane distance determiner 603, tilt orientationdeterminer 605, plane of view corrector 607, pen scribing surfacecoordinate determiner 609 and pen display location determiner 611 is/arehereby expressly defined to include a tangible computer readable storagedevice or storage disk such as a memory, a digital versatile disk (DVD),a compact disk (CD), a Blu-ray disk, flash (non-volatile) memory etc.storing the software and/or firmware. Further still, the example EO penlocation determiner 113 of FIG. 6 may include elements, processes and/ordevices in addition to, or instead of, those illustrated in FIG. 6,and/or may include more than one of any or all of the illustratedelements, processes and devices.

A flowchart representative of example machine readable instructions forimplementing the example EO pen location determiner 113 of FIG. 6 isshown in FIG. 7. In this example, the machine readable instructionscomprise a program for execution by a processor such as the processors1201 and 1312 shown in the example processor platforms 1200 and 1300discussed below in connection with FIGS. 12 and 13. The program may beembodied in software stored on a tangible computer readable storagemedium such as a CD-ROM, a floppy disk, a hard drive, a digitalversatile disk (DVD), a Blu-ray disk, or a memory (flash memory)associated with the processor 1312, but the entire program and/or partsthereof could alternatively be executed by a device other than theprocessor 1312 and/or embodied in firmware or dedicated hardware.Further, although the example program is described with reference to theflowchart illustrated in FIG. 7, many other methods of implementing theexample EO pen location determiner apparatus 113 may alternatively beused. For example, the order of execution of the blocks may be changed,and/or some of the blocks described may be changed, eliminated, orcombined.

As mentioned above, the example processes of FIGS. 7-11 may beimplemented using coded instructions (e.g., computer and/or machinereadable instructions) stored on a tangible computer readable storagemedium such as a hard disk drive, a flash memory, a read-only memory(ROM), a compact disk (CD), a digital versatile disk (DVD), a cache, arandom-access memory (RAM), nonvolatile flash memory and/or any otherstorage device or storage disk in which information is stored for anyduration (e.g., for extended time periods, permanently, for briefinstances, for temporarily buffering, and/or for caching of theinformation). As used herein, the term tangible computer readablestorage medium is expressly defined to include any type of computerreadable storage device and/or storage disk and to exclude propagatingsignals and to exclude transmission media. As used herein, “tangiblecomputer readable storage medium” and “tangible machine readable storagemedium” are used interchangeably. Additionally or alternatively, theexample processes of FIGS. 7-11 may be implemented using codedinstructions (e.g., computer and/or machine readable instructions)stored on a non-transitory computer and/or machine readable medium suchas a hard disk drive, a flash memory, a read-only memory, a compactdisk, a digital versatile disk, a cache, a random-access memory and/orany other storage device or storage disk in which information is storedfor any duration (e.g., for extended time periods, permanently, forbrief instances, for temporarily buffering, and/or for caching of theinformation). As used herein, the term non-transitory computer readablemedium is expressly defined to include any type of computer readablestorage device and/or storage disk and to exclude propagating signalsand to exclude transmission media. As used herein, when the phrase “atleast” is used as the transition term in a preamble of a claim, it isopen-ended in the same manner as the term “comprising” is open ended.

FIG. 7 is a flowchart of a method for determining the location of an EOpen. The program of FIG. 7 begins at block 701. Referring to FIG. 7,image accessor 601 (FIG. 6) accesses a first image of a scribing surfacethat is imaged from a first perspective and a second image of thescribing surface that is imaged from a second perspective (block 701).In some examples, the first image has a first field of view and thesecond image has a second field of view. In some examples, the firstfield of view and the second field of view at least partially overlap todefine a stereoscopic dot-sample plane of view. Image accessor 601accesses the first and the second images from first and second imagersin real time. In some examples, information from the plane of viewprovides image information from the scribing surface that is used todetermine pen location.

Pen-to-scribing-plane distance determiner 603 (FIG. 6) determines adistance from a pen to a scribing plane at a plurality of points (block703). The determining of the distance from the pen to the scribing planeat the plurality of points produces a plurality of distancemeasurements. In some examples, pen-to-scribing-plane distancedeterminer 603 determines a distance from a pen to the scribing plane ata plurality of points by cross-correlating corresponding dots (e.g.,marks, etc.) in the plane of view of the images that are imaged by firstimager 109 and second imager 111. In some examples,pen-to-scribing-plane distance determiner 603 uses stereographic imageprocessing to process images of the dots to determine the distance tothe scribing plane from the pen at several points as discussedhereinabove. In some examples, pen-to-scribing-plane distance determiner603 determines the distance to the scribing plane from the pen atseveral points by determining an overlap region of the first and thesecond fields of view, identifying dots in the overlap region anddetermining the distance to a plurality of dots in the overlap region.Pen-to-scribing-plane distance determiner 603 uses triangulation todetermine the distance or depth z of each of a plurality of dots. Insome examples pen-to-scribing-plane distance determiner 603 can use theformula b*f/(x1−x2) to determine the distance z to each of a pluralityof dots, where x1 and x2 correspond to image points associated withimager L and imager R of FIG. 5 (it should be noted that y1 and y2 alsoassociated with imager L and R are not shown in FIG. 5). In otherexamples, other formulas can be used to determine the distance z to eachof the plurality of dots.

Pen-to-scribing-plane distance determiner 603 provides this informationto the tilt orientation determiner 605 such that it can be used todetermine the tilt orientation of the scribing plane.

Tilt orientation determiner 605 determines the tilt orientation of thescribing plane relative to the pen based on the plurality of distancemeasurements performed by pen-to-scribing-plane distance determiner 603(block 705). In some examples, tilt orientation determiner 605determines the tilt orientation of the scribing plane relative to thepen by determining the three dimensional orientation of the scribingplane based on the distances to the plurality of dots. In some examples,three dots are used to calculate the orientation of the dot plane tohigh accuracy. In other examples, other numbers of dots are used.

In some examples, tilt orientation determiner 605, based on the distancez to each of the plurality of dots, determines the three-dimensionalcoordinates of the dots provided by pen-to-scribing-plane distancedeterminer 603 using equations such as the following:z=f*b/(x1−x2)=b/dx=x1*z/f or b+x2*z/fy=y1*z/f or y2*z/f

In other examples, other equations and/or manner of determining thecoordinates of the dots can be used.

As described above, tilt orientation determiner 605, based on thecoordinates of the dots, determines the three-dimensional orientation ofthe dot plane/scribing surface. For example, for dots/points P, Q and Rhaving coordinates P(1,−1,3), Q(4,1,−2) and R(−1,−1,1) tilt orientationdeterminer 605 can determine the orientation of the dot plane/scribingsurface by determining the plane that passes through those points. Insome examples, to find the plane passing through points P, Q and R, thetilt orientation determiner 605 determines the cross product of a vector“a” from P to Q and a vector “b” from Q to R to obtain a vectororthogonal to the plane. In the preceding example, a=<3,2,−5> andb=<−5,−2,3>, thus:

${\overset{\rightarrow}{n} = {{\overset{\rightarrow}{a} \times \overset{\rightarrow}{b}} = {{\begin{matrix}\hat{i} & \hat{j} & \hat{k} \\3 & 2 & {- 5} \\{- 5} & {- 2} & 3\end{matrix}} = {< {\text{-}4}}}}},{16},{4 >}$

The equation of the plane is −4x+16y+4z=−32. The manner of determiningtilt orientation described herein is an example. In other examples,other manner of determining tilt orientation can be used.

Plane of view corrector 607 corrects the plane of view associated witheither the first perspective or the second perspective based on acomparison of a depth of field associated with the first perspective anda depth of field associated with the second perspective (block 707). Insome examples, plane of view corrector 607 corrects the plane of viewassociated with either the first perspective or the second perspectiveby accessing the scribing plane orientation information and based onthis information determining whether the first or the second imager hasthe best depth of field. The imager that is determined to have the bestdepth of field is dependent upon the orientation of the EO pen at thepoint in time at which the determination is made. Other criteria such asdiscussed previously, can include but is not limited to image contenterror such as smudged or missing dot/marker, which can be used fordetermination. Plane of view corrector 607 corrects the plane of viewusing the depth of field or other criteria of the imager that isdetermined to have the best depth of field or other criteria.

Pen scribing surface coordinate determiner 609 determines thecoordinates of the EO pen on the EO pen scribing surface based on thecorrected plane of view (block 709). In some examples, the gridorientation marks from both imagers are used to provided directionalityand basis for incremental separation of superimposed reference gridlines. Based on the perspective corrected dot locations, and a referencegrid, a decoding/determination of the location of the EO pen tip isperformed.

Pen display location determiner 611 determines the position of the EOpen on a display (display 117 in FIG. 1) based on the pen coordinateinformation accessed from pen scribing surface coordinate determiner 609(block 711). In some examples, the EO pen nib/tip location can becommunicated back to the host for display via Bluetooth wirelessconnection.

FIG. 8 is a flowchart of block 703 of FIG. 7 where the distance from theEO pen to the scribing plane at a plurality of points is determined.Referring to FIG. 7, pen-to-scribing-plane distance determiner 603determines the overlap region of the first and second fields of view(block 801). Pen-to-scribing-plane distance determiner 603 identifiesdots in the overlap region of the first and the second fields of view(block 803) and pen-to-scribing-plane distance determiner 603 determinesthe distance to dots in the overlap region of the first and the secondfields of view (block 805).

FIG. 9 is a flowchart of block 705 of FIG. 7 where the tilt orientationof the scribing plane relative to the EO pen is determined. Referring toFIG. 9, tilt orientation determiner 605 accesses the distance to each ofthe dots in the scribing plane (block 901). Tilt orientation determiner605 determines the three-dimensional orientation of the scribing planebased on the distance to each dot (block 903).

FIG. 10 is a flowchart of block 903 of FIG. 9 where the threedimensional orientation of the scribing plane is determined based on adetermined distance to each of a plurality of dots. Referring to FIG.10, tilt orientation determiner 605 accesses the determined distances zto each of the plurality of dots (block 1001).

Tilt orientation determiner 605, based on the distance z to each of theplurality of dots, determines the three-dimensional coordinates of thedots (block 1003). In some examples, the three-dimensional coordinatesof the dots can be determined using equations such as the following:z=f*b/(x1−x2)=b/dx=x1*z/f or b+x2*z/fy=y1*z/f or y2*z/f

In other examples, other equations and/or manner of determining thecoordinates of the dots can be used.

Tilt orientation determiner 605, based on the coordinates determined atblock 1003, determines the three-dimensional orientation of the dotplane/scribing surface (block 1005). For example, for dots/points P, Qand R having coordinates P(1,−1,3), Q(4,1,−2) and R(−1,−1,1) the tiltorientation determiner 605 can determine the orientation of the dotplane/scribing surface by determining the plane that passes throughthose points. In some examples, to find the plane passing through pointsP, Q and R the tilt orientation determiner 605 determines the crossproduct of a vector “a” from P to Q and a vector “b” from Q to R toobtain a vector orthogonal to the plane. In one example, a=<3,2,−5> andb=<−5,−2,3>, thus:

${\overset{\rightarrow}{n} = {{\overset{\rightarrow}{a} \times \overset{\rightarrow}{b}} = {{\begin{matrix}\hat{i} & \hat{j} & \hat{k} \\3 & 2 & {- 5} \\{- 5} & {- 2} & 3\end{matrix}} = {< {\text{-}4}}}}},{16},{4 >}$

Based on the above, the equation of the plane is −4x+16y+4z=−32. Themanner of determining tilt orientation described herein is an example.In other examples, other manners of determining tilt orientation can beused.

FIG. 11 is a flowchart of block 707 of FIG. 7 where the plane of viewassociated with the first or second perspective is corrected. Referringto FIG. 11, plane of view corrector 607 accesses scribing planeorientation information (block 1101). Plane of view corrector 607determines if a first imager of the first and second imagers has thebest depth of field (block 1103). If it is determined that a firstimager that is assessed for depth of field of the first and secondimagers has the best depth of field, the plane of view is correctedusing the first imager of the first and second imagers (block 1105). Ifit is determined that a first imager that is assessed for depth of fielddoes not have the best depth of field, the plane of view is correctedusing the second imager (block 1107).

FIG. 12 is a block diagram of an example processor platform 1200 capableof executing the instructions of FIGS. 7-11 to implement the apparatusof FIG. 6. Referring to FIG. 12, the processor platform includessystem-on-chip multi-core vision processor 1201. In some examples,processor 1201 can include EO pen location determiner 113 whoseoperation is described in detail herein and is not repeated here. Insome examples, processor 1201 can include a dual CSI-2 input (e.g.,Movidius MA 2150 or Inuitive NU4000) SoC processor. In other examples,other types of processors can be used. Referring to FIG. 12, processor1201 is coupled to LED control circuitry 1203. In some examples, LEDcontrol circuitry controls the operation of LED 1205 such as bycontrolling the emission of light onto a scribing surface (e.g., 104 inFIG. 1). In some examples, processor 1201 can have input components thatcan include force sensor 1207, IMU sensor 1209, buttons 1211 and haptics1213. In addition, processor 1201 can be communicatively coupled toBluetooth I/O 1215 and flash memory 1217. The power provided toprocessor 1201 is controlled by power on/off charging circuit 1219 thatis coupled to battery 1221. In operation, processor 1201 processesimager control instructions that control the operation of imagers 1223and 1225 and the image data that is acquired by imagers 1223 and 1225such as from the scribing surface. In some examples, processor 1201 is amulti-core processor that is designed for three-dimensional dataprocessing. In some examples, the core processors are backed by hardwareaccelerators for depth, simultaneous localization and mapping (SLAM) andComputer Vision processing which reduces latency.

FIG. 13 is a block diagram of another example processor platform 1300capable of executing the instructions of FIGS. 7-11 to implement theapparatus of FIG. 6. The processor platform 1300 can be, for example, anEO pen embedded microprocessor, microcontroller, system-on-chip (SoC) oralternatively a server, a personal computer, a mobile device (e.g., acell phone, a smart phone, a tablet such as an iPad™), a personaldigital assistant (PDA), an Internet appliance, a DVD player, a CDplayer, a digital video recorder, a Blu-ray player, a gaming console, apersonal video recorder, a set top box, or any other type of computingdevice.

The processor platform 1300 of the illustrated example includes aprocessor 1312. The processor 1312 of the illustrated example ishardware. For example, the processor 1312 can be implemented byintegrated circuits, logic circuits, microprocessors or controllers fromany desired family or manufacturer.

The processor 1312 of the illustrated example includes a local memory1313 (e.g., a cache). The processor 1312 of the illustrated example isin communication with a main memory including a volatile memory 1314 anda non-volatile memory 1316 via a bus 1318. The volatile memory 1314 maybe implemented by Synchronous Dynamic Random Access Memory (SDRAM),Dynamic Random Access Memory (DRAM), RAMBUS Dynamic Random Access Memory(RDRAM) and/or any other type of random access memory device. Thenon-volatile memory 1316 may be implemented by flash memory and/or anyother desired type of memory device. Access to the main memory 1314,1316 is controlled by a memory controller.

The processor platform 1300 of the illustrated example also includes aninterface circuit 1320. The interface circuit 1320 may be implemented byany type of interface standard, such as an I2C, wireless Bluetooth,wireless WiFi, Ethernet interface, a universal serial bus (USB), and/ora PCI express interface.

In the illustrated example, input devices 1322 are connected to theinterface circuit 1320. The input device(s) 1322 permit(s) a user toenter data and commands into the processor 1312. The input device(s) canbe implemented by, for example, an audio sensor, a microphone, a camera(still or video), an accelerometer, an inertial measurement unit, a pennib pressure sensor, and/or a voice recognition system.

Output devices 1324 are also connected to the interface circuit 1320 ofthe illustrated example. The output devices 1324 can be implemented, forexample, by display devices (e.g., a light emitting diode (LED), anorganic light emitting diode (OLED), a liquid crystal display, a cathoderay tube display (CRT), a touchscreen, a tactile output device, aprinter and/or speakers). The interface circuit 1320 of the illustratedexample, thus, may include a graphics driver card, a graphics driverchip or a graphics driver processor.

The interface circuit 1320 of the illustrated example also includes acommunication device such as a transmitter, a receiver, a transceiver, amodem and/or network interface card to facilitate exchange of data withexternal machines (e.g., computing devices of any kind) via a network1326 (e.g., an Ethernet connection, a digital subscriber line (DSL), atelephone line, coaxial cable, a cellular telephone system, etc.) ornon-network connection.

The processor platform 1300 of the illustrated example also includesmass storage devices 1328 for storing software and/or data. Examples ofsuch mass storage devices 1328 include floppy disk drives, hard drivedisks, compact disk drives, Blu-ray disk drives, RAID systems, anddigital versatile disk (DVD) drives.

The coded instructions 1332 of FIGS. 7-11 may be stored in the massstorage device 1328, in the volatile memory 1314, in the non-volatilememory 1316, and/or on a removable tangible computer readable storagemedium such as a CD or DVD.

From the foregoing, it will be appreciated that methods and apparatushave been disclosed which provide for a higher accuracy (position andtilt), greater depth-of-field, and the ability to apply a more highlyefficient computational pipe line for decoding EO pen position withvarying pen to scribing surface orientation changes. Some examples usestereoscopic imaging that enables a more deterministic and simplerrather than more computationally intensive and stochastic or randomprocess to determine scribing plane to pen orientation. Some examplesuse embedded logic. Some examples also allow for significantly higherposition measurement and data update rates from the EO pen to a hostcomputer (desktop, notebook, tablet, phone, etc.) via an RF channel(e.g., Bluetooth). In some examples, if hardware acceleration via a DSP(ASIC or FPGA) or other computationally accelerated core is utilizedimprovements in EO pen performance can be significantly increased incomparison to the speed of other solutions due to the describedincreased computational efficiency.

Some EO pen systems use a much physically larger sized imager (e.g., 25times larger) and hence require an extended optical path with a foldmirror between the imaging lens and a CMOS imager. Some EO pen systemsuse imager pixel sizes that are greater than 15 um square. Some exampleimagers use imager pixels of sizes that are less than 5 um square (e.g.,3×3 um). Some examples use the highly compact CameraCube™ technologyavailable from OmniVision™. Examples described herein include astraightforward design and require uncomplicated optical alignmentprocedures during manufacture.

Although certain example methods, apparatus and articles of manufacturehave been disclosed herein, the scope of coverage of this patent is notlimited thereto. On the contrary, this patent covers all methods,apparatus and articles of manufacture fairly falling within the scope ofthe claims of this patent.

What is claimed is:
 1. An apparatus comprising: a distance determiner todetermine a distance of a pen to a scribing surface based on a firstimage output by a first imager of the pen and a second image output by asecond imager of the pen, the first image associated with a first planeof view and the second image associated with a second plane of view; atilt orientation determiner to determine a tilt orientation of thescribing surface relative to the pen; a corrector to adjust at least oneof the first plane of view or the second plane of view based on the tiltorientation of the scribing surface; and a location determiner todetermine a location of the pen relative to the scribing surface basedon the at least one of the adjusted first plane of view or the adjustedsecond plane of view.
 2. The apparatus of claim 1, wherein the tiltorientation is a three-dimensional orientation of the scribing surfaceand the distance determiner is to: determine a first distance of the penfrom a first point in the first plane of view of the first image, thefirst distance corresponding to a first depth of the first point; anddetermine a second distance of the pen from a second point in the secondplane of view of the second image, the second distance corresponding toa second depth of the second point, the tilt orientation determiner todetermine the three-dimensional orientation of the scribing surfacebased on the first distance and the second distance.
 3. The apparatus ofclaim 2, wherein the first plane of view and the second plane of view atleast partially overlap to define an overlap region and the first pointand the second point are in the overlap region.
 4. The apparatus ofclaim 1, wherein the corrector is to: determine a first depth of fieldof the first imager based on the tilt orientation; determine a seconddepth of field of the second imager based on the tilt orientation;perform a comparison of the first depth of field and the second depth offield; and correct the first plane of view based on the comparison. 5.The apparatus of claim 1, further including a coordinate determiner todetermine coordinates of the pen relative to the scribing surface basedon the at least one of the adjusted first plane of view or the adjustedsecond plane of view, the location determiner to determine the locationof the pen based on the coordinates.
 6. A system comprising: a stylusincluding a first imager and a second imager, the first imager to outputa first image of a scribing surface from a first perspective and thesecond imager to output a second image of the scribing surface from asecond perspective; and one or more processors to: determine a pluralityof distance measurements, respective ones of the distance measurementsrepresenting a distance of the stylus from the scribing surface at arespective point in the first image or the second image; determine atilt orientation of the scribing surface relative to the stylus based ontwo or more of the plurality of distance measurements; adjust a plane ofview associated with at least one of the first perspective or the secondperspective based on the tilt orientation; and identify a location ofthe stylus relative to the scribing surface based on coordinate data forthe stylus associated with the adjusted plane of view.
 7. The system ofclaim 6, wherein the first imager has a first field of view and thesecond imager has a second field of view, the first field of view atleast partially overlapping the second field of view.
 8. The system ofclaim 6, wherein the one or more processors is to adjust the plane ofview associated with the first perspective and generate a firstcorrected image based on the adjusted plane of view.
 9. The system ofclaim 8, wherein the one or more processors is to determine thecoordinate data from the first corrected image.
 10. The system of claim8, wherein the one or more processors is to: determine a first depth offield associated with the first imager based on the tilt orientation;and select the plane of view associated with the first perspective foradjustment based on the first depth of field.
 11. A non-transitorycomputer readable storage medium comprising instructions that, whenexecuted, cause at least one machine to at least: determine a distanceof a pen to a display surface based on a first image output by the penand a second image output by the pen, the first image associated with afirst plane of view and the second image associated with a second planeof view; determine a tilt orientation of the display surface relative tothe pen; adjust at least one of the first plane of view or the secondplane of view based on the tilt orientation of the display surface; anddetermine a location of the pen relative to the display surface based onthe at least one of the adjusted first plane of view or the adjustedsecond plane of view.
 12. The non-transitory computer readable storagemedium of claim 11, wherein the instructions, when executed, cause theat least one machine to: determine a first distance of the pen from afirst point in the first plane of view of the first image, the firstdistance corresponding to a first depth of the first point; determine asecond distance of the pen from a second point in the second plane ofview of the second image, the second distance corresponding to a seconddepth of the second point; and determine the tilt orientation of thedisplay surface based on the first distance and the second distance, thetilt orientation corresponding to a three-dimensional orientation of thedisplay surface.
 13. The non-transitory computer readable storage mediumof claim 12, wherein the first plane of view and the second plane ofview at least partially overlap to define an overlap region and thefirst point and the second point are in the overlap region.
 14. Thenon-transitory computer readable storage medium of claim 11, wherein theinstructions, when executed, cause the at least one machine to:determine a first depth of field of the pen based on the tiltorientation; determine a second depth of field of the pen based on thetilt orientation; perform a comparison of the first depth of field andthe second depth of field; and correct the first plane of view based onthe comparison.
 15. The non-transitory computer readable storage mediumof claim 11, wherein the instructions, when executed, cause the at leastone machine to: determine coordinates of the pen relative to the displaysurface based on the at least one of the adjusted first plane of view orthe adjusted second plane of view; and determine the location of the penbased on the coordinates.
 16. An apparatus comprising: memory;instructions; and processor circuitry to execute the instructions to:determine a distance of a pen to a display surface based on a firstimage output by the pen and a second image output by the pen, the firstimage associated with a first plane of view and the second imageassociated with a second plane of view; determine a tilt orientation ofthe display surface relative to the pen; adjust at least one of thefirst plane of view or the second plane of view based on the tiltorientation of the display surface; and determine a location of the penrelative to the display surface based on the at least one of theadjusted first plane of view or the adjusted second plane of view. 17.The apparatus of claim 16, wherein the processor circuitry is to:determine a first distance of the pen from a first point in the firstplane of view of the first image, the first distance corresponding to afirst depth of the first point; determine a second distance of the penfrom a second point in the second plane of view of the second image, thesecond distance corresponding to a second depth of the second point; anddetermine the tilt orientation of the display surface based on the firstdistance and the second distance, the tilt orientation corresponding toa three-dimensional orientation of the display surface.
 18. Theapparatus of claim 17, wherein the first plane of view and the secondplane of view at least partially overlap to define an overlap region andthe first point and the second point are in the overlap region.
 19. Theapparatus of claim 16, wherein the processor circuitry is to: determinea first depth of field of the pen based on the tilt orientation;determine a second depth of field of the pen based on the tiltorientation; perform a comparison of the first depth of field and thesecond depth of field; and correct the first plane of view based on thecomparison.
 20. The apparatus of claim 16, wherein the processorcircuitry is to: determine coordinates of the pen relative to thedisplay surface based on the at least one of the adjusted first plane ofview or the adjusted second plane of view; and determine the location ofthe pen based on the coordinates.